Sensing device for medical facilities

ABSTRACT

A medical system may utilize a modular and extensible sensing device to derive a two-dimensional (2D) or three-dimensional (3D) human model for a patient in real-time based on images of the patient captured by a sensor such as a digital camera. The 2D or 3D human model may be visually presented on one or more devices of the medical system and used to facilitate a healthcare service provided to the patient. In examples, the 2D or 3D human model may be used to improve the speed, accuracy and consistency of patient positioning for a medical procedure. In examples, the 2D or 3D human model may be used to enable unified analysis of the patient&#39;s medical conditions by linking different scan images of the patient through the 2D or 3D human model. In examples, the 2D or 3D human model may be used to facilitate surgical navigation, patient monitoring, process automation, and/or the like.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Provisional U.S. PatentApplication No. 62/941,203, filed Nov. 27, 2019, the disclosure of whichis incorporated herein by reference in its entirety.

BACKGROUND

The quality of healthcare services provided to a patient largely dependson the amount of information available to the patient and medicalprofessionals such as doctors rendering the services. For example, withradiation therapy and medical imaging, success often hinges upon theability to place and maintain a patient in a desirable position so thatthe treatment or scan can be performed in a precise and accurate manner.Having up-to-date knowledge about the patient's physical characteristics(e.g., height, body shape, pose, etc.) in these situations may offermany benefits including, for example, faster and more accuratepositioning of the patient in accordance with a scan or treatmentprotocol, less manual work, more consistent results, etc. The knowledgeabout the patient's physical characteristics can also be used toaggregate different medical records of the patient (e.g., based oncommon attributes of the patient present in the medical records and/orthe identity of the patient), and derive a comprehensive view of thepatient's diagnostic and treatment history.

In other example situations such as during a surgical procedure,information about a patient's physiques, movements, and/or positions mayoffer insight and guidance for both treatment planning and execution.The information may be utilized, for instance, to locate and navigatearound a treatment site of the patient. When visually presented inreal-time, the information may also provide means for monitoring thestate of the patient during the procedure.

With the advancement of technologies in areas such as computer visionand artificial intelligence, it is desirable to utilize thesetechnologies to acquire patient information in an automatic andreal-time manner, and to improve the quality of healthcare servicesutilizing the acquired information.

SUMMARY

Described herein are systems, methods and instrumentalities forproviding healthcare services to a patient using a medical system. Themedical system may comprise a modular and extensible sensing devicecapable of generating, in real-time, a two-dimensional (2D) orthree-dimensional (3D) human model and a representation thereof for apatient based on at least one image of the patient captured by one ormore sensors. The sensing device may comprise or be coupled to thesesensors. The sensing device may comprise one or more processorsconfigured to receive the at least one image of the patient from thesensors. In response to receiving the at least one image, the sensingdevice (e.g., the one or more processors of the sending device) mayanalyze the image to extract a plurality of features (e.g., featurevectors) that is representative of one or more anatomicalcharacteristics of the patient and estimate the 2D or 3D human model ofthe patient based on the features. The 2D or 3D human model of thepatient may include, for example, a parametric human body model and therepresentation of the 2D or 3D model may include a 2D or 3D mesh of thepatient.

Once generated, the 2D or 3D human model of the patient and/or itsrepresentation may be transmitted to one or more other devices of themedical system, for example, together with the at least one image of thepatient captured by the one or more sensors. The 2D or 3D human modeland/or its representation may be used to improve one or more aspects ofthe healthcare services provided to the patient including, for example,patient positioning, patient monitoring, scan image unification andanalysis, surgical navigation, etc.

The one or more sensors described herein may include a digital camera, ared-green-blue (RGB) sensor, a depth sensor, a RGB plus depth (RGB-D)sensor, a thermal sensor such as infrared (FIR) or near-infrared (NIR)sensor, etc. As such, the at least one image of the patient captured bythe sensors may include a photo of the patient captured by the digitalcamera or an RGB or thermal image captured by a corresponding sensor.

The sensing device may be modular and extensible and may comprise one ormore slots each including at least one of a power connector or acommunication interface circuit. Each of the slots may be configured tohost a respective set of sensors or processors configured to work with arespective imaging modality or a respective patient. The communicationinterface circuit may be configured to transmit or receive informationon behalf of the respective sensors or processors hosted in therespective slot. The power connector may be configured to provide powerto the respective set of sensors or processors hosted in the respectiveslot. As such, the sensing device may be capable of working withdifferent types of downstream devices and/or different applicationrequirements, for example, by increasing or decreasing the number ofsensors or processors included in the sensing device.

The sensing device may be calibrated with other devices in the medicalsystem. For example, the one or more processors of the sensing devicemay be configured to determine a spatial relationship between a firstcoordinate system associated with the sensing device and a secondcoordinate system associated with a medical device such as a medicalscanner. This way, the representation of the 2D or 3D human model of thepatient generated by the sensing device may be used together with scanimages obtained from other imaging modalities (e.g., to align the scanimages with the representation of the human model), thus enablingunified analysis of the patient's medical records obtained fromdifferent sources. The 2D or 3D human model or its representation mayalso be used to improve the speed, accuracy and consistency of patientpositioning relating to a medical procedure, to facilitate surgicalnavigation and patient monitoring, etc.

BRIEF DESCRIPTION OF THE DRAWINGS

A more detailed understanding of the examples disclosed herein may behad from the following description, given by way of example inconjunction with the accompanying drawing.

FIG. 1 is a simplified block diagram illustrating an example medicalsystem as described herein.

FIG. 2 is a simplified diagram illustrating how a sensing device asdescribed herein may be facilitate the operation of a medical system.

FIG. 3 is a simplified block diagram illustrating an example sensingdevice as described herein.

FIG. 4 is a flow diagram illustrating the operations of a sensing deviceas described herein.

FIG. 5 is a flow diagram illustrating the operations of a medical systemas described herein.

DETAILED DESCRIPTION

The present disclosure is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings.

FIG. 1 is a diagram illustrating an example system 100 for providinghealthcare services in a medical environment or at a medical facility,such as in a hospital. The healthcare services may include, for example,a medical scan or imaging procedure conducted using a medical scanner102 (e.g., a computer tomography (CT) scanner, a magnetic resonanceimaging (MRI) machine, a positron emission tomography (PET) scanner, anX-ray machine, etc.), a radiation treatment procedure delivered througha medical linear accelerator (LINAC) (not shown), or a surgicalprocedure performed in an operating room. The system 100 may include atleast one sensing device 104 (e.g., an image capturing device)configured to capture images of a patient 106 (or an object 106) in oraround the medical environment (e.g., in front of the medical scanner102, lying on a scan or treatment bed, etc.). In examples, the sensingdevice 104 may be configured to detect the presence of the patient 106using a sensor such as a motion, radar, or light sensor, and capture theimages of the patient in response to detecting the patient. In examples,the sensing device 104 may not be configured to automatically detect thepresence of the patient 106, but may be controlled (e.g., manuallycontrolled, remotely controlled, programmed, etc.) to capture the imagesof the patient, for example, in response to receiving a control signalinstructing the sensing device to do so.

In examples, the sensing device 104 may comprise one or more sensorsincluding one or more cameras (e.g., digital color cameras, 3D cameras,etc.), one or more red, green and blue (RGB) sensors, one or more depthsensors, one or more RGB plus depth (RGB-D) sensors, one or more thermalsensors such as infrared (FIR) or near-infrared (NIR) sensors, one ormore motion sensors, one or more radar sensors, and/or other types ofimage capturing circuitry configured to generate images (e.g., 2D imagesor photos) of a person, object or scene. Depending on the type ofsensors or image capturing circuits used, the images generated by thesensing device 104 may include, for example, one or more photos of thepatient taken by a camera, one or more thermal images of the patientgenerated by a thermal sensor, one or more radar images of the patientproduced by a radar sensor, and/or the like. The sensors of the sensingdevice 104 may be configured to capture the images of the patient,object or scene in response to detecting the patient, object or scene,based on a preconfigured schedule or time interval, or upon receiving acontrol signal triggering the image capturing.

In examples, the sensing device 104 may be configured to becommunicatively coupled to sensors (e.g., cameras) that already exist inthe medical system 100 (e.g., sensors external to the sensing device104), and to receive and process the images captured by these sensors asif the images are captured by internal sensors of the sensing device104. For instance, the medical system 100 may already include sensorsconfigured for surveillance, diagnostic or treatment purposes. Thesesensors may be capable of capturing images of a patient, an object or ascene present in the medical system 100 in response to detecting thepatient, object or scene, based on a preconfigured schedule or timeinterval, or upon receiving a control signal triggering the imagecapturing. The sensors may also have the capability to transmit thecaptured images (e.g., raw imagery data comprising pixel information) toa receiving device in the medical system 100, for example, through acommunication interface. In at least these scenarios, the sensing device104 may be added to the medical system 100, e.g., as a receiving deviceof the images produced by the sensors, and obtain information from theimages that may be used to improve the quality of services provided bythe medical system 100, as described herein. The sensing device 104 maytransmit the information obtained from the images (e.g., in the form a2D or 3D human model or a representation thereof) to a downstream deviceor application, for example, together with the imagery data originallyreceived from the sensors.

The sensing device 104 may be installed or placed in various locationsof the medical system 100 such as inside a scan room, inside a treatmentroom, inside an operation room, around a registration desk, in ahallway, on the medical scanner 102, on a ceiling, near a doorway, on awall, etc. From these locations, the sensing device 104 may captureimages of a patient, an object or a scene that is in the field of view(FOV) of the sensing device (e.g., from a certain viewpoint or viewingangle). The FOV of the sensing device 104 may be adjusted (e.g.,manually or automatically by sending a control signal to the sensingdevice) so that multiple images may be taken from different viewpointsor viewing angles.

In examples, the sensing device 104 may be a portable or mobile device,in which case the sensing device may be placed or hosted (e.g., placedon a shelf, attached to a hanging mount, etc.) at the various locationsdescribed above and may be moveable from one location to another. Inexamples, the sensing device 104 may be physically connected to (e.g.,be wired together with) a location or another device of the medicalsystem 100, in which case the sensing device may operate as an integralpart of the connected device. And although only one sensing device 104is depicted in FIG. 1, the medical system 100 may include multiplesensing devices each located at a respective location and having arespective FOV.

The sensing device 104 may include a functional unit configured toprocess the images generated by the sensors described herein (e.g.,sensors comprised in the sensing device and/or preexisting sensors thatare external to the sensing device). The functional unit may be coupledto the sensors (e.g., via a wired or wireless communication link) andconfigured to receive images from the sensors (e.g., via a pushmechanism). The functional unit may also be configured to retrieveimages from the sensors (e.g., via a pull mechanism), for example, on aperiodic basis or in response to receiving a control signal instructingthe functional unit to retrieve the images. In examples, the functionalunit may be configured to receive a notification from the sensors whenan image has become available and to retrieve the image in response toreceiving the notification.

The images captured by the sensors may include two-dimensional (2D) orthree-dimensional (3D) images depicting a patient, an object or a scenepresent in a medical environment. Each of the 2D or 3D images maycomprise a plurality of pixels, lines, and/or vertices. The functionalunit may be configured to analyze these images (e.g., at a pixel level)and generate a 2D or 3D model (e.g., a parametric model such as onebased on a skinned multi-person linear (SMPL) model) of the patient,object or scene depicted in the images, for example, using a neuralnetwork (e.g., a convolutional neural network). The 2D or 3D modelgenerated by the functional unit may include one or more 2D keypoints,one or more 3D keypoints, one or more parameters (e.g., a set of 72shape and/or pose parameters) for constructing the model, and/or otherinformation relating to a 2D or 3D representation of the patient, objector scene. The 2D or 3D model may be represented, for example, by one ormore of a 2D mesh, a 3D mesh, a 2D contour, a 3D contour, etc. toindicate the pose, shape and/or other anatomical characteristics of apatient and thereby to facilitate a plurality of downstream medicalapplications and services for the patient including, for example,patient positioning, medical protocol design, unified or correlateddiagnoses and treatments, medical environment monitoring, surgicalnavigation, etc. For ease of description, when a 2D or 3D human model ofa patient or a 2D or 3D model of an object or scene is referred toherein, it should be interpreted to include not only the model itselfbut also a representation of the model in any graphical or visual form.

In examples, the sensing device 104 may function as an edge device(e.g., with limited computation and/or storage capacities), and may passone or more computation and/or storage tasks (e.g., all computationand/or storage tasks) to a server device. The server device may be anetwork-based (e.g., cloud-based) server device and may be configuredto, upon completing a computation task requested by the sensing device104, provide the computation results (e.g., a recovered human 3D meshmodel) to other devices of the medical system 100 including the sensingdevice 104 for further processing and/or delivery.

The sensing device 104 may include a communication circuit configuredexchange information with one or more other devices of the medicalsystem 100, for example, over a communication network 108. Thecommunication network 108 may be a wired or a wireless network, or acombination thereof. For example, the communication network 108 may beestablished over a public network (e.g., the Internet), a privatenetwork (e.g., a local area network (LAN), a wide area network (WAN)),etc.), a wired network (e.g., an Ethernet network), a wireless network(e.g., an 802.11 network, a Wi-Fi network, etc.), a cellular network(e.g., a Long Term Evolution (LTE) or 5G network), a frame relaynetwork, a virtual private network (VPN), a satellite network, and/or atelephone network. The communication network 108 may include one or morenetwork access points. For example, the communication network 108 mayinclude wired and/or wireless network access points such as basestations and/or internet exchange points through which one or morecomponents of the medical system 100 may be connected to exchange dataand/or other information. Such exchange may utilize routers, hubs,switches, server computers, and/or any combination thereof.

The communication circuit of the sensing device 104 may be configured toreceive imagery data produced by the sensors described herein and maytransmit, e.g., directly from the sensing device 104 and/or via an edgeserver, one or more data streams carrying the 2D or 3D human model of apatient or object generated by the sensing device to a receiving device.In addition to the 2D or 3D model, the one or more data streams may alsoinclude the original imagery data (e.g., raw images comprising pixels ofinformation) produced by the sensors. The original imagery data mayinclude, for example, RGB data produced by a RGB sensor, depth dataproduced by a depth sensor, RGB plus depth (RGB-D) data from a RGB-Dsensor, infrared (IR) data from an IR sensor, etc. The original imagerydata may be used by a downstream device or application for variouspurposes including, e.g., verifying, reconstructing, or refining the 2Dor 3D human model generated by the sensing device, comparing orcombining the imagery data with other medical images or scans, etc.

The medical system 100 may include a programming device 110 configuredto configure and/or control one or more of the medical scanner 102 orthe sensing device 104. For example, the programming device 110 may beconfigured to initialize and modify one or more operating parameters ofthe medical scanner 102 or the sensing device 104 such as a resolutionat which an image is captured, a frequency for data exchange to and fromthe sensing device 104 (e.g., frequency for image transmission orretrieval), a frame or bit rate associated with the data exchange, aduration of data storage on the sensing device, etc. The programmingdevice 110 may also be configured to control one or more aspects of theoperation of the medical scanner 102 or the sensing device 104including, e.g., triggering a calibration operation for the devices,providing calibration parameters such as those relating to the spatialrelationship between different coordinate systems to the devices,adjusting the direction or orientation of a sensor, zooming in orzooming out a sensor, triggering a system reset, etc. The programmingdevice 110 may include a mobile device such a smart phone, a tablet, ora wearable device. The programming device 110 may include a desktopcomputer, a laptop computer, and/or the like. The programming device 110may be configured to communicate with the medical scanner 102 and/or thesensing device 104 over the communication network 108. The programmingdevice 110 may receive information and/or instructions from a user(e.g., via a user interface implemented on the programming device), andsend the received information and/or instructions to the medical scanner102 or the sensing device 104 via the communication network 108.

The medical system 100 may further include a processing device 112configured to receive one or more data streams from the sensing device104 and/or a device performing computation or storage tasks on behalf ofthe sensing device 104. The processing device may be co-located with thesensing device 104 (e.g., in a same room) or be located remotely fromthe sensing device 104 (e.g., in a control room or processing centeraway from where the sensing device 104 is located). The processingdevice 112 may be communicatively coupled to other devices in themedical system 100 including the sensing device 104, The processingdevice 112 may comprise a general-purpose computer, a special-purposecomputer, one or more computation and storage units, a cloud-basedplatform, and/or the like. Merely by way of example, a cloud-basedplatform referred to herein may include a private cloud, a public cloud,a hybrid cloud, a community cloud, a distributed cloud, an inter-cloud,a multi-cloud, and/or the like. Further, although the processing device112 is depicted in FIG. 1 as a separate device from the medical scanner102, the applicability of the examples provided herein is not limited tosuch a configuration. For example, the processing device 112 may be apart of (e.g., comprised in) the medical scanner 102 and may still becapable of performing the various functions described herein.

The one or more data streams received by the processing device 112 mayinclude a 2D or 3D model of a patient, an object, or a scene generatedby the sensing device 104 and/or imagery data (e.g., raw or originalimagery data) produced by a sensor. As described herein, the 2D or 3Dmodel may be generated by the sensing device 104 or a server deviceperforming computation or storage tasks on behalf of the sensing device104. Once received, the 2D or 3D model may be used by the processingdevice 112 to assist with various aspects of healthcare services. Forexample, the processing device 112 may also be coupled to a repository114 configured to store patient medical records including basic patientinformation, scan images of the patients obtained through other imagingmodalities (e.g., CT, MR, X-ray, SPECT, PET, etc.) of the medical system100 at different times, diagnostic and/or treatment history of thepatients, etc. The processing device 112 may retrieve all or a subset ofthe medical records for a patient from the repository 114 and analyzethe retrieved medical records in conjunction with the 2D or 3D model ofthe patient provided by the sensing device 104. The processing device112 may also receive (e.g., in real-time) a scan image of the patientproduced by the medical scanner 102, and analyze the received scan imagein conjunction with the 2D or 3D model of the patient provided by thesensing device 104. For instance, the processing device 112 may receivea scan image of the patient from the repository 114 or the medicalscanner 102, align the scan image with the 2D or 3D human model of thepatient, and render the aligned image and 2D or 3D model visually (e.g.,in an overlaid picture) to allow the scan image to be presented andanalyzed with reference to anatomical characteristics (e.g., body shapeand/or pose) of the patient as indicated by the model. This way, moreinsight may be gained into the organ(s) or tissue(s) of the patientcaptured in the scan image based on the additional information providedby the 2D or 3D model. For instance, the 2D or 3D model may indicate aposition of the patient at the time the scan image is taken.

The alignment described above may be accomplished, for example, byidentifying and matching respective anatomical landmarks (e.g., jointlocations) in the scan image and the 2D or 3D model. The alignment mayalso be accomplished, for example, by determine a correlation (e.g., aspatial relationship) between a first coordinate system associated withthe 2D or 3D model (e.g., with the sensing device 104) and a secondcoordinate system associated with a medical environment or a medicaldevice, and use the correlation to match up corresponding areas of thescan image and the 2D or 3D model. The correlation between the twocoordinate systems may be determined, for example, during system setupbased on markers placed in the medical environment and/or by comparingsample images produced by the sensing device 104 and a concerned medicaldevice (e.g., such as the medical scanner 102).

Using the 2D or 3D model as a common reference, the processing device112 may be able to align multiple different scan images (e.g., fromrespective imaging modalities) of the patient together, for example, byaligning each scan image with the 2D or 3D model and thereafter aligningone scan image with another using the 2D or 3D model as an intermediatereference. When referred to herein, the alignment of two or more scanimages or the alignment of a scan image with the 2D or 3D model mayinclude overlaying one scan image with another scan image or overlayingthe 2D or 3D model with the scan image.

By establishing a correlation between a 2D or 3D human model of apatient produced by the sensing device 104 and scan images of thepatient obtained from other sources or modalities, the processing device112 may determine a target scan or treatment area of a patient andindicate the target area to the patient or a medical professionaladministering a medical procedure for the patient. For example, theprocessing device may determine, based on preexisting scan imagesobtained from the other sources or modalities, that an organ of thepatient (e.g., lungs, head, etc.) needs to be further scanned ortreated. The processing device 112 may align one or more of thepreexisting scan images with a 2D or 3D model of the patient (e.g., a 3Dmesh of the patient) produced by the sensing device 104 and determinewhere the organ (e.g., the target scan area) is on the 3D mesh. Theprocessing device 112 may indicate the determined target area to thepatient or the medical professional visually, for example, by markingthe target area on the 3D mesh. This way, localization of a target scanor treatment site may be accomplished in real time (e.g., while thepatient is waiting to be scanned or treated), non-invasively, and/orwithout using radioactive tracers, and the results may be used toprovide guidance to treatment planning, protocol design, and/or surgicalnavigation (e.g., to guide a surgical robot such as a built-inmultiple-degrees-of-freedom (MDOF) robot or MDOF robotic arms, forcollision avoidance, etc.).

Based on a 2D or 3D human model of a patient produced by the sensingdevice 104, the processing device 112 may also be configured to identifya background object in a scan image of the patient, and segment orsubtract the background object from the scan image so that the scanimage may be analyzed without interference. For example, a scan image ofthe patient received by the processing device 112 (e.g., from themedical scanner 102) may include a scan bed in the background of thescan image. To segment or subtract the area or pixels of the scan imagethat correspond to the scan bed, the processing device 112 may determinea silhouette of the patient based on a 2D or 3D human model of thepatient produced by the sensing device 102 while the patient is lying onthe scan bed, and identify the area or pixels of the scan image that lieoutside the silhouette as the area of pixels corresponding to the scanbed. The processing device 112 may then segment or subtract the area orpixels corresponding to the scan bed from the scan image and present theimage resulting from the segmentation or subtraction for furtheranalysis.

The processing device 112 may also utilize the 2D or 3D human model of apatient produced by the sensing device 104 to facilitate positioning ofthe patient during a scan or treatment procedure. For example, asdescribed herein, the 2D or 3D human model of the patient may include a3D mesh or contour of the patient and/or parameters that indicate aplurality of anatomical characteristics of the patient (e.g., the bodyshape of the patient, a pose of the patient, and/or a distance of thepatient from the medical scanner 102, etc.). Based on these indications,the processing device 112 may determine, e.g., in real time (e.g., whilethe patient is still in the position or pose indicated by the 2D or 3Dhuman model), whether the position or pose of the patient meets apredetermined protocol for the scan or treatment procedure. If theprocessing device determines that the position or pose of the patientdoes not meet the predetermined protocol, the processing device 112 mayprovide instructions (e.g., a command) to the patient and/or the medicalprofessional administering the procedure for correcting the position orpose of the patient. The instructions may include visual and/or audioinstructions including, for example, animations and/or presentationssuitable for augmented reality (AR) devices. This way, the patientand/or the medical professional may visualize the differences betweenthe patient's current position and a desired position (e.g., a guidelineposition) and make necessary adjustment to minimize or eliminate thedifferences.

The anatomical characteristics of the patient indicated by the 2D or 3Dhuman model may also be used to determine a radiation consumption levelof the patient. For example, the processing device 112 may estimate theheight and/or weight of the patient as well as the distance between thepatient and the medical scanner 102 based on the 2D or 3D human model ofthe patient, and use the estimation together with operating parametersof the medical scanner (e.g., strength of the magnetic field generatedby the scanner, amount of energy released by the scanner, etc.) tocalculate the amount radiation received by the patient. The processingdevice 112 may generate a report of the radiation level and/or providean alert if the level exceeds a certain threshold.

The 2D or 3D human model of the patient produced by the sensing device102 may also be used to determine an identity of the patient. Forexample, the processing device 112 may comprise or be coupled to afeature database comprising known visual features (e.g., keypointsindicating joint locations, joint angles, facial features, body shapes,positions, poses, walking patterns, etc.) of one or more patients. Theseknown visual features or keypoints may be pre-computed and stored in thefeature database. In response to receiving a 2D or 3D human model of thepatient from the sensing device 102, the processing device 112 may matchthe visual features or keypoints comprised in the 2D or 3D human modelwith those stored in the feature database and determine the identity ofthe patient based on a matching score (e.g., higher scores indicatebetter matching).

Additionally or alternatively, the processing device 112 may utilizeartificial neural networks trained for visual recognition to determinethe identity of the patient. In examples, the neural networks mayinclude a convolutional neural network (CNN) that comprises a cascade oflayers each trained to make pattern matching decisions based on arespective level of abstraction of the visual characteristics containedin a set of images (e.g., in the pixels of the images). The training ofthe neural network may be performed using large amounts of imagery dataand/or specific loss functions through which the neural network maylearn to extract features (e.g., in the form of feature vectors) from anewly provided input image, determine whether the features match thoseof a known person, and indicate the matching results at an output of theneural network. Using one or more such neural networks, the processingdevice 112 may be configured to compare visual features of a patientindicated by or extracted from the 2D or 3D human model against thoseextracted from other images of the patient and determine the identity ofthe patient based on the comparison or matching.

Once determined, the identity of the patient may be used to personalizethe healthcare services provided to the patient. For example, theprocessing device 112 may use the patient's identity to retrieve medicalrecords associated with the patient from the repository 114, unify orsynthesize the retrieved medical records with other information obtainedabout the patient (e.g., physical and/or anatomical characteristics ofthe patient indicated by the 2D or 3D human model), and present theunified or synthesized records to allow a more comprehensive review oranalysis of the patient medical conditions.

As described herein, the processing device 112 may be communicativelycoupled to other medical devices of the medical system 100 (e.g., themedical scanner 102). As such, the processing device 112 may beconfigured to control one or more of the other medical devices based oninformation acquired from the 2D or 3D human model of the patientproduced by the sensing device 104. For example, upon determining thephysical characteristics of the patient based on the 2D or 3D humanmodel of the patient, the processing device 112 may transmit a commandor control signal (e.g., to the medical scanner 102 or another devicecontrolling the medical scanner 102) to adjust an operating parameter ofthe medical scanner 102, e.g., to better accommodate the patient. Suchan operating parameter may be associated with, for example, the heightof a scan bed, a scan angle, a dosage level, a position or orientationof the medical scanner, etc., and the command or control signal may betransmitted in digital and/or analog forms.

Further, although examples are provided herein for generating a 2D or 3Dhuman model of a patient and using the model to improve the quality ofhealthcare services for the patient, it will be appreciated that thesensing device 104 may receive images of multiple patients and generaterespective 2D or 3D human models (e.g., representations of the 2D or 3Dmodels) for the patients based on these received images, for example,simultaneously. The images of the patients may be captured by a samesensor or by different sensors, and the generation of the 2D or 3D humanmodels may be accomplished by a same functional unit or by differentfunctional units. The sensing device 104 may be configured to be modularand/or extensible so that sensors and/or processors (e.g., GPUs) may beadded to or removed from the sensing device to accommodate varyingapplication requirements. The modularity and extensibility of thesensing device will be described in greater detail below.

The sensing device 104 may also be configured to receive images of anobject and a scene present in a medical environment, analyze the imagesusing the techniques described herein, and generate a 2D or 3D models ofthe object or scene or a report regarding the object or scene that mayindicate a condition of the medical environment. For example, based onimages of one or more objects captured by a sensor, the sensing device104 may determine respective categories of the one or more objects and anumber of items in each of the categories, wherein the categories mayinclude at least one of medical tools, medicine, or food supplies. Thesensing device 104 may transmit the information (e.g., in a report or avisual representation) to the processing device 112, which may use theinformation to automate or improve the efficiency of facility managementsuch as inventory management, tool tracking, traffic control, etc. Inexamples, the information obtained via the sensing device 104 mayindicate the condition in an operating room (e.g., the state of surgicaltools or devices, an amount of blood loss of a patient, etc.) so thatcritical issues may be identified and addressed. In examples, theinformation obtained via the sensing device 104 may indicate that foodor medical supply inventory at a location is running low so thatreplenishment may be ordered. In examples, the information obtained viathe sensing device 104 may indicate that traffic in an area of themedical environment is heavy so that patients and medical staff may beinformed to avoid the area.

The medical system 100 and/or the processing device 112 may comprise adisplay device 116 and/or an audio device (not shown), which may beconfigured to display the various commands, instructions, alerts and/orreports generated by the processing device 112 and/or other devices inthe system, as described herein. The display device 116 may include oneor more monitors (e.g., computer monitors, TV monitors, tablets, mobiledevices such as smart phones, etc.), one or more speakers, one or moreaugmented reality (AR) devices (e.g., AR goggles), and/or otheraccessories configured to facilitate audio or visual presentations. Thedisplay device 116 may be communicatively coupled to the processingdevice 112 and/or the sensing device 104 via the communication network108 or another suitable communication link. As described herein, theinformation or instructions presented via the display device 116 mayinclude desired positions and poses of a patient for a medicalprocedure, positions taken by the patient during past scans, adjustmentinstructions for the patient to get into the desired positions or poses,surgical navigation instructions, dosage consumption levels, etc. Theinformation and/or instructions may be presented to the patient 106 invarious formats including, for example, audios, videos, animations, ARpresentations, etc.

The systems, methods and instrumentalities described herein may befurther illustrated by FIG. 2, which shows an example medicalenvironment (e.g., a scan room 200) in which a sensing device (e.g., thesensing device 104 shown in FIG. 1) may be installed. As shown, the scanroom 200 may be equipped with a medical scanner 202 (e.g., the medicalscanner 102 shown in FIG. 1) such as an upright X-ray scanner and adisplay device 216 (e.g., the display device 116 shown in FIG. 1). Thedisplay device 216 may be communicatively coupled to the medical scanner202 (e.g., via a wired or wireless communication network) or be a partof the medical scanner 202. A sensing device (not shown in FIG. 2) suchas the sensing device 104 of FIG. 1 may be added to the scan room 200(e.g., installed on a wall, attached to a hanging mount off the ceiling,placed on a shelf, etc.) and configured to generate a 2D or 3D humanmodel such as a 3D mesh of a patient 206 while the patient is standingin front of the medical scanner 101. As described herein, the 2D or 3Dhuman model may be generated based on one or more images (e.g., photosor thermal images) of the patient captured by a sensor (e.g., a camera,a thermal sensor, etc.) comprised in the sensing device or by one ormore sensors already installed in the scan room 202 (e.g., existingsensors that are external to the sensing device). A functional unit(e.g., which may comprise one or more processors) of the sensing devicemay receive these images of the patient, analyze the images to determinea plurality of features that is representative of one or more anatomicalcharacteristics (e.g., joint locations, joint angles, etc.) of thepatient, and estimate the 2D or 3D human model of the patient based onthe plurality of features. The 2D or 3D human model may indicate the oneor more physical characteristics of the patient such as the body shapeand/or pose of the patient, position of the patient relative to themedical scanner 202, and/or other anatomical features of the patient.

The 2D or 3D human model may be estimated and transmitted in real time(e.g., while the patient is standing in front of the medical scanner101) in one or more data streams to a processing device (e.g., theprocessing device 112 shown in FIG. 1), which may be located inside thescan room 200 or remotely from the scan room 200. The transmission maybe performed, for example, via a communication circuit of the sensingdevice, and, in addition to the 2D or 3D human model, the one or moredata streams may also include the images of the patient (e.g., raw imagedata such as pixel data) produced by the sensors. In response toreceiving the 2D or 3D human model generated by the sensing deviceand/or the images produced by the sensors, the processing device mayrender the 2D or 3D human model and/or the images of the patient in away that allows the information to be used to improve various aspects ofa healthcare service provided to the patient 206 including, e.g.,unified or coordinated medical diagnosis, guided imaging or surgicalprocedures, patient positioning and/or monitoring, medical facilitymanagement, and/or the like. For example, the processing device mayoverlay an image of the patient with the 2D or 3D human model (e.g., a3D mesh) of the patient, as shown in FIG. 2, so that a target scan siteof the patient may be determined and the medical scanner may becontrolled (e.g., remotely and/or automatically) to focus on the targetscan site (e.g., to collimate X-ray beams towards the target scan siteto reduce radiation, improve subject contrast and image quality, etc.).As another example, the 2D or 3D human model may be used to identify,detect, and/or track the movements and activities of a patient or anobject in the medical environment for purposes of process monitoring,process optimization, resource pre-allocation, resource utilizationanalysis, automated process logging, workflow analysis and optimization,automated process cost code estimation, etc.

FIG. 3 illustrates an example sensing device 300 (e.g., the sensingdevice 104 shown in FIG. 1) that may be placed or installed in a medicalsystem such as the medical system 100 described herein. The sensingdevice 300 may comprise a sensor 302, a functional unit 304, and a powersupply that are configured to be hosted in a housing. Although twosensors are shown in the figure, it will be appreciated that the sensingdevice 300 may comprise any number of sensors, or the sensing device 300may not comprise any sensor and may instead be configured to receiveimages from an external sensor (e.g., an existing sensor in a medicalsystem). Further, although one or more of the components are shown inFIG. 3 as being inside or outside of the functional unit 304, thesecomponents are not restricted to the configuration shown in the figureand may be moved inside or outside of the functional unit 304 withoutaffecting the functionalities of the sensing device described herein.

As described herein, the sensor 302 may include a RGB sensor, a depthsensor, a RGB plus depth (RGB-D) sensor, a thermo sensor such as a FIRor NIR sensor, a radar sensor, a motion sensor, a camera (e.g., adigital camera) and/or other types of image capturing circuitryconfigured to generate images (e.g., 2D images or photos) of a person,object, and/or scene in the FOV of the sensor. And the images generatedby the sensor 302 may include, for example, one or more photos, thermalimages, and/or radar images of the person, object or scene. Each of theimages may comprise a plurality of pixels that collectively represent agraphic view of the person, object or scene and that may be analyzed toextract features that are representative of one or more characteristicsof the person, object or scene.

The sensor 302 may be communicatively coupled to the functional unit304, for example, via a wired or wireless communication link. Inexamples, the sensor 302 may be configured to transmit images generatedby the sensor to the functional unit 304 (e.g., via a push mechanism).In examples, the functional unit 304 may be configured to retrieveimages from the sensor 302 (e.g., via a pull mechanism). Thetransmission and/or retrieval may be performed on a periodic basis(e.g., based on a preconfigured schedule) or in response to receiving acontrol signal triggering the transmission or retrieval. Such a controlsignal may be sent, for example, by the sensor 302, e.g., when an imagehas become available, or by a remote control device such as a mobiledevice or a system controller, e.g., upon receiving an input from auser.

The sensor 302 may be configured to receive one or more control signals(e.g., digital control messages) from the functional unit 304 that mayaffect the operation of the sensor 302. For example, the sensor 302 mayreceive a command from the functional unit 304 to adjust the FOV of thesensor (e.g., by manipulating a direction or orientation of the sensor).As another example, the sensor 302 may receive a command from thefunctional unit 304 that changes the resolution at which the sensortakes images of a person, object or scene.

The sensor 302 and/or the functional unit 304 (e.g., one or morecomponents of the functional unit 304) may be powered by the powersupply 306, which may comprise an alternative current (AC) power sourceor a direct current (DC) power source (e.g., a battery power source).When a DC power source such as a battery power source is used, the powersupply 306 may be rechargeable, for example, by receiving a chargingcurrent from an external source via a wired or wireless connection. Forexample, the charging current may be received by connecting the sensingdevice 300 to an AC outlet via a charging cable and/or a chargingadaptor (including a USB adaptor). As another example, the chargingcurrent may be received wirelessly by placing the sensing device 300into contact with a charging pad.

The functional unit 304 may comprise one or more of a communicationinterface circuit 308, a data processing unit 310, a computation unit312, a data rendering unit 314, a memory 316, or a programming and/orcalibration application programming interface (API) 318. It should benoted that the architecture shown in FIG. 3 is provided merely as anexample and is not meant to limit the scope of the disclosure to such anarchitecture. For example, the functional unit 304 is not restricted toincluding the exact components as shown in FIG. 3. Two or more of thecomponents (e.g., functionalities of the components) may be combined,any one of the components may be divided into sub-components, any one ofthe components may be omitted, more components may be added, etc. Assuch, even though the functionalities of the sensing device 300 aredescribed herein as being associated with respective one or more of thecomponents, it will be appreciated that those functionalities may alsobe performed by a different component and/or be divided among multipleother components.

In the example shown in FIG. 3, the functional unit 304 may beconfigured to receive or retrieve images of a patient from the sensor302 via the communication interface circuit 308, which may include oneor more wired and/or wireless network interface cards (NICs) such asethernet cards, WiFi adaptors, mobile broadband devices (e.g., 4G/LTE/5Gcards or chipsets), etc. In examples, a respective NIC may be designatedto communicate with a respective sensor. In examples, a same NIC may bedesignated to communication with multiple sensors.

The images received or retrieved from the sensor 302 may be provided tothe data processing unit 310, which may be configured to analyze theimages and estimate (e.g., construct or recover) models based on theimages to depict (e.g., mathematically and/or visually) one or morecharacteristics (e.g., body shape, pose, etc.) of the patient depictedin the images. For example, the data processing unit 310 may beconfigured to analyze at least one of the images produced by the sensor302 (e.g., at a pixel level), identify a plurality of features that isrepresentative of one or more anatomical or physical characteristics ofthe patient, and estimate parameters (e.g., mesh parameters) that may beused to construct a human model (e.g., a parametric human model) of thepatient. In examples, the plurality of features may represent jointlocations and/or joint angles of the patient as depicted in the at leastone image produced by the sensor 302, and the human model may include aSMPL model defined by a plurality of parameters that indicates one ormore characteristics of the patient. The parameters estimated by thedata processing unit 310 may include one or more shape parameters, β,and/or one or more pose parameters, θ. The shape parameters may comprisecoefficients of a principal component analysis (PCA) space that may beused to determine (e.g., recover) a blend shape of the patient. The poseparameters may be derived based on the locations and/or angles of aplurality of joints of the patient recovered from the at least one image(e.g., 23 joints comprised in a skeletal rig as well as a root joint),and may indicate a pose of the patient. Based on the shape and/or poseparameters (e.g., a set of 72 parameters corresponding to the 23joints), the data processing unit 310 may determine a plurality of meshvertices that may be used to generate a representation (e.g., a 3D mesh)of the 2D or 3D human model constructed for the patient. In addition,the data processing unit 310 may also be configured to use theshape/pose parameters and/or the features extracted from the at leastone image to determine an identity of the patient. The data processingunit 310 may indicate the identity of the patient to other devices orcomponents of the medical system to personalize the healthcare servicesprovided to the patient.

The functionality of the data processing unit 310 may be facilitated bythe computation unit 312, which may be configured to perform variouscomputation intensive tasks such as feature extraction and/or featureclassification based on the images produced by the sensor 302. Thecomputation unit 312 may comprise one or more neural networks such asone or more convolutional neural networks (CNNs) and/or one or more deepneural networks (DNNs) trained for visual recognition. The neuralnetworks may comprise multiple layers (e.g., an input layer, one or moreconvolutional layers, one or more non-linear activation layers, one ormore pooling layers, one or more fully connected layers, and/or anoutput layer). Each of the layers may correspond to a plurality offilters (e.g., kernels) and each filter may be designed to detect a setof keypoints that collectively represent a respective visual feature orpattern. The filters may be associated with respective weights that,when applied to an input, produce an output indicating whether certainvisual features or patterns have been detected. The weights associatedwith the filters may be learned by the neural networks through atraining process that comprises inputting a large number of images fromone or more training datasets to the neural network (e.g., in a forwardpass), calculating losses resulting from weights currently assigned tothe filters (e.g., based on a loss function such as a margin based lossfunction), and updating (e.g., in a backward pass) the weights assignedto the filters so as to minimize the losses (e.g., based on stochasticgradient descent). Once trained, the neural networks may take an imageat the input layer, extract and/or classify visual features or patternsfrom the image (e.g., in the form of feature vectors), and provide anindication at the output layer for whether an extracted feature matchesa known feature and/or whether the extracted feature falls within aspecific category or class.

In addition to or in lieu of the neural networks described above, thecomputation unit 312 may comprise or may be coupled to a featuredatabase configured to store a plurality of known features of thepatient (e.g., facial features, body shapes, body contours, jointlocations, joint angles, walking patterns, poses, etc.). Each featuremay correspond a combination of keypoints arranged in a specific mannerin the images such as points at which the direction of the boundary ofan object changes abruptly, intersection points between two or more edgesegments, etc. The keypoints may be characterized by well-definedpositions in the image space and/or stability to illumination orbrightness perturbations. Accordingly, these keypoints may be identifiedbased on image derivatives, edge detection, curvature analysis, and/orthe like. And once identified, the keypoints and/or the featurerepresented by the keypoints may be described with a feature descriptoror feature vector. In an example implementation of such featuredescriptors or vectors, information related to the feature (e.g.,appearance of the local neighborhood of each keypoint) may berepresented by (e.g., encoded into) a series of numerical values storedin the feature descriptors or vectors. The descriptors or vectors maythen be used as “fingerprints” for differentiating one feature fromanother or for matching one feature with another.

The one or more human models (e.g., parameters for constructing thehuman models) generated by the data processing unit 310 and/or thecomputation unit 312 may be provided to the data rendering unit 314,which may be configured to generate representations (e.g., 2D or 3Drepresentations such as 2D or 3D meshes) of the human models that depictone or more anatomical or physical characteristics of the patient. Forexample, the data rendering unit 314 may receive a plurality of meshvertices determined by the data processing unit 310 and/or thecomputation unit 312. Each of the vertices may in turn includerespective position, normal, texture, and/or shading information. Basedon these vertices, the data rendering unit 314 may create a 2D or 3Dmesh of the patient, for example, by connecting multiple vertices withedges to form a polygon (e.g., such as a triangle), connecting multiplepolygons to form a surface, using multiple surfaces to determine a 3Dshape, and applying texture and/or shading to the surfaces and/orshapes. Once created, the 2D or 3D representation may be output by thedata render unit 314 in one or more data streams to a receiving device(e.g., the processing device 112 in FIG. 1), for example, over thecommunication interface circuit 308. Further, in addition to the humanmodel information generated by the data processing unit 310 and/or thecomputation unit 312, the data rendering unit 314 may also be configuredto receive the original imagery data produced by the sensor 302 andoutput the original imagery data to the receiving device, for example,in a same or different data stream as the data stream carrying therepresentation.

Each of the data processing unit 310, the computation unit 312, or thedata rendering unit 314 may comprise one or more processors such as acentral processing unit (CPU), a graphics processing unit (GPU), amicrocontroller, a reduced instruction set computer (RISC) processor, anapplication specific integrated circuit (ASIC), an application-specificinstruction-set processor (ASIP), a physics processing unit (PPU), adigital signal processor (DSP), a field programmable gate array (FPGA),or a combination thereof. The data processing unit 310, computation unit312, and/or data rendering unit 314 may also comprise other type(s) ofcircuits or processors capable of executing the functions describedherein. Further, the data processing unit 310, the computation unit 312,or the data rendering unit 314 may utilize the memory 316 to facilitateone or more of the operations described herein. For example, the memory316 may include a machine-readable medium configured to store dataand/or instructions that, when executed, cause the processing unit 310,the computation unit 312, or the data rendering unit 314 to perform oneor more of the functions described herein. Examples of amachine-readable medium may include volatile or non-volatile memoryincluding but not limited to semiconductor memory (e.g., electricallyprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM)), flash memory, and/or the like.And even though not shown in FIG. 3, the sensing device 300 may alsocomprise one or more mass storage devices that include a magnetic disksuch as an internal hard disk, a removable disk, a magneto-optical disk,a CD-ROM or DVD-ROM disk, etc., on which instructions and/or data may bestored to facilitate the performance of the functions described herein.

The sensing device 300 and/or the functional unit 304 may be configuredto be modular and extensible such that sensors, communication circuits,data processing units, computation units, and/or data rendering unitsmay be added to or removed from the sensing device 300, for example, toaccommodate different system settings, configurations and/orrequirements in a medical environment (e.g., the medical system 100).For example, if output quality is the priority in the medicalenvironment, a high-resolution sensor (e.g., a high-resolution camera)may be included in (e.g., added to) the sensing device 300 to meet thepriority. On the other hand, if the priority is on output speed (e.g.,frame rate), a sensor (e.g., a camera) with lower resolution and/or acommunication circuit with faster bitrates (e.g., an ethernet cardrather than a WiFi card) may be used to meet the output requirement. Asanother example, the sensing device 300 may be configured to work (e.g.,simultaneously) with multiple devices in the medical environment such asmultiple imaging modalities (e.g., CT, MR, etc.), in which case thesensing device may include respective sets of sensors, communicationcircuits, power supplies, processors (e.g., data processing units,computation units, and/or data rendering units as described herein) forthe respective medical devices. As yet another example, the sensingdevice 300 may be configured to receive images of multiple patients(e.g., from different sensors) and generate respective 2D or 3D modelsfor the patients based on the images, for example, simultaneously. Insuch a scenario, the sensing device may include respective sets ofsensors, communication circuits, power supplied, processors (e.g., dataprocessing units, computation units, and/or data rendering units asdescribed herein) for capturing and processing the respective images ofthe respective patients.

In examples, the sensing device 300 and/or the functional unit 304 maycomprise multiple slots (e.g., expansion boards, etc.) each equippedwith at least one of a power connector or a communication circuit (e.g.,a network interface card, a USB port, etc.) capable of transmitting andreceiving information over a wired or wireless communication link.Sensors and/or processors (e.g., data processing units, computationunits, and/or data rendering units as described herein) may be hosted in(e.g., inserted into) these slots, upon which the sensors and/orprocessors may receive power through the respective power connectors andperform data exchange with one or more internal or external devices viathe respective communication circuits. These sensors and processors mayrespectively possess similar capabilities as the sensor 302, the dataprocessing unit 310, the computation unit 312, and the data renderingunit 314 described herein, and may be added to or removed from thesensing device 300, for example, to accommodate changing conditionsand/or requirements in the medical environment in which the sensingdevice 300 is installed.

For example, the sensing device 300 may include a first set of one ormore sensors configured to capture images of a first patient (e.g., apatient taking a X-ray scan) and a first set of one or more processors(e.g., the data processing unit 310, the computation unit 312, and thedata rendering unit 314) configured to generate a first 2D or 3D modelfor the first patient and provide the model to a first receiving device(e.g., a controller associated with the X-ray scanner). The first set ofone or more sensors and/or the first set of one or more processors maybe hosted in a first slot of the sensing device 300, which may providepower and/or communication service to the sensors and/or processors. Thesensing device 300 may also include a second slot configured to host(e.g., provide power and/or communication service to) a second set ofone or more sensors and/or a second set of one or more processors (e.g.,units similar to the data processing unit 310, the computation unit 312,and the data rendering unit 314). Such a second set of sensors may beconfigured to capture images of a second patient (e.g., a patient takinga CT scan) and the second set of one or more processors may beconfigured to generate a second 2D or 3D model for the second patientand provide the model to a second receiving device (e.g., a controllerassociated with the CT scanner). In this manner, the sensing device 300may be modular and extensible to handle data processing tasks associatedwith different patients and/or imaging modalities. In other examplesituations such as when the amount of computation, communication, and/ordata storage workload approaches or exceeds the capabilities of one setof sensors and/or processors, more of the sensors and/or processors maybe added to share the workload.

The operation of the sensing device 300 may be configured and/orcontrolled through the programming/calibration API 318, for example,using a remote programming device such as the programming device 110 inFIG. 1. In examples, the programming/calibration API 318 may beconfigured to receive commands (e.g., one or more digital messages) fromthe programming device that change the operating parameters of thesensing device 300 such as the orientation and/or FOV of a sensor, aresolution at which a sensor captures images, a quality required of arepresentation of a 2D or 3D model of a patient generated by the sensingdevice, a periodicity at which images are received or retrieved from asensor, a bit rate at which the sensing device transmits a 2D or 3Dhuman model of a patient and/or the original imagery data captured by asensor, etc. In response to receiving a command from the programmingdevice, the sensing device 300 (e.g., the functional unit 304) mayadjust one or more aspects of its operation in accordance with thecommand. For instance, if the command specifies a higher output quality,the sensing device 300 may output a high-resolution mesh in response,and if the command specifies a higher frame rate, the sensing device 300may output a lower-resolution mesh, but at an increased frame rate.

The sensing device 300 (e.g., the functional unit 304) may also beconfigured to receive ad hoc commands through theprogramming/calibration API 318. Such ad hoc commands may include, forexample, a command to zoom in or zoom out a sensor, a command to resetthe sensing device 300 (e.g., restart the device or reset one or moreoperating parameters of the device to default values), a command for thesensing device 300 to transmit or re-transmit certain types of data suchas the meta-data relating to a human mesh generated by the sensingdevice (e.g., estimated parameters for constructing the human mesh) to areceiving device, a command to enable or disable a specificfunctionality of the sensing device 300 such as whether the sensingdevice should attempt to determine the identity of a patient, etc. Thesensing device 300 (e.g., the functional unit 304) may also beprogrammed and/or trained (e.g., over a network) via theprogramming/calibration API 318. For example, the sensing device 300 mayreceive training data and/or operating logics through theprogramming/calibration API 318 during and/or after an initialconfiguration process.

The sensing device 300 (e.g., the functional unit 304) may be calibratedwith the medical environment in which the sensing device is installedand/or with one or more other devices in the medical environment such asthe medical scanner 102 in the medical system 100. The calibration maybe performed, for example, during initial configuration of the sensingdevice 300 and/or in response to receiving a calibration command via theprogramming/calibration API 318. The calibration may include determininga relationship (e.g., spatial relationship) between a first coordinatesystem associated with the sensing device 300 and a second coordinatesystem associated with the medical environment or a medical device inthe medical environment (e.g., such as the medical scanner 102 in FIG.1). In examples, the sensing device 300 (e.g., the functional unit 304)may be configured to determine (e.g., learn) the spatial relationshipbetween the first and second coordinate systems based on an offsetbetween the respective origins of the two coordinate systems and/or arotation angle between respective X or Y axes of the two coordinatesystems. The sensing device 300 may receive information regarding suchan offset and/or rotational angle via a configuration message (e.g.,transmitted by a programming device). In examples, the sensing device300 may be configured to learn the offset and/or the rotational angle bycomparing sample images generated by the sensing device and the medicalscanner, for example, based on annotated or marked areas of the images.In examples, the sensing device 300 may be configured to learn theoffset and/or the rotational angle based on markers placed in themedical environment such as one or more objects placed in the corners ofa scan room (e.g., during a calibration process).

Once the spatial relationship (e.g., spatial correlation) between thefirst and second coordinate systems are determined, the sensing device300 and/or other devices in the medical system may utilize the spatialrelationship for human model recovery, scan image analysis, and/or thelike. For example, the sensing device 300 may receive an image of apatient from a sensor (e.g., a camera) that includes a scan bed in thebackground of the image, and the sensing device 300 may have knowledgeabout the location of the scan bed in the scan room as defined by thecoordinate system of the scan room (e.g., the sensing device may havelearned or been given the location of the scan bed during systemconfiguration). If the sensing device 300 can determine the spatialrelationship between the coordinate system of the scan room and thecoordinate system of the sensor that captures the image, the sensingdevice 300 may convert the location of the scan bed in the formercoordinate system (e.g., associated with the scan room) to a location inthe latter coordinate system (e.g., associated with the sensing deviceor the sensor), for example, using a transformation matrix, based on theoffset and/or rotational angle described herein, etc. The sensing device300 may then be able to segment the scan bed from the image such that a2D or 3D model may be generated just for the patient (e.g., excludingthe scan bed from the model).

The spatial relationship (e.g., spatial correlation) between a firstcoordinate system associated with the sensing device 300 (e.g., a sensorof the sensing device) and a second coordinate systems associated with amedical device (e.g., the medical scanner 102 in FIG. 1) may also beused by a processing device or controller associated with the medicaldevice to process or analyze medical information collected for a patientvia the medical device together with a 2D or 3D human model of thepatient generated by the sensing device 300. For example, based on thespatial correlation between the respective coordinate systems associatedwith the sensing device 300 and the medical scanner 102, a processingdevice or controller associated with the medical scanner 102 may be ableto project a scan image of the patient captured by the medical scanner102 onto the 2D or 3D human model of the patient generated by thesensing device 300 to allow for unified analysis of the scan image, asdescribed herein.

Although the description of the sensing device 300 is provided usingimages of a patient as an example, it will be appreciated that similartechniques can also be used by the sensing device 300 to process imagesof an object or a scene. As described herein, information extracted fromimages of an object or scene may be used for various facility managementpurposes in a medical environment including, for example, inventorymanagement, tool tracking, traffic control, facilitate monitoring,and/or the like.

FIG. 4 is a flow diagram illustrating example operations that may beperformed by a sensing device (e.g., the sensing device 104 in FIG. 1 orthe sensing device 300 in FIG. 3) as described herein. The sensingdevice may be configured to start the example operations at 402periodically, for example, based on a predetermined time interval, uponreceiving a command to start the operations, or in response to a patientbeing detected in medical environment. At 404, the sensing device mayreceive one or more images of the patient captured by a sensor such asan RGB sensor, a thermal sensor, or a digital camera. As describedherein, the sensor may be an existing sensor (e.g., an existing camera)in the medical environment or may be a sensor comprised in the sensingdevice. At 406, the sensing device may analyze the received images andextract a plurality of features (e.g., in the form of feature vectors)that is representative of one or more anatomical characteristics of thepatient as depicted in the one or more images. The features mayrepresent, e.g., joint locations and/or joint angles of the patient.

Based on the extract features, the sensing device may determine (e.g.,estimate) a set of parameters relating to a human model of the patientat 408, for example, by recovering a shape of the patient and/or aplurality of joint angles of the patient based on the extractedfeatures. The set of parameters may include, for example, one or moreshape parameters that collectively indicate a body shape of the patientand one or more pose parameters that collectively indicate a pose of thepatient. Utilizing the shape and/or pose parameters (e.g., a set of 72parameters corresponding to 23 joints of the patient), the sensingdevice may create a representation (e.g., a 2D or 3D mesh) of the humanmodel at 410, for example, by determining a plurality of vertices of amesh associated with the human model and create the mesh using thevertices. At 412, the sensing device may transmit the human model (e.g.,the mesh) and/or the images received at 404 to a receiving device. Theset of operations of the sensing device may then end at 414.

For simplicity of explanation, the operations of the sensing device aredepicted in FIG. 4 and described herein with a specific order. It shouldbe appreciated, however, that these operations may occur in variousorders, concurrently, and/or with other operations not presented ordescribed herein. Furthermore, not all illustrated operations may berequired to be performed by the sensing device.

FIG. 5 is a flow diagram illustrating example operations that may beperformed by one or more devices of a medical system (e.g., the medicalsystem 100 in FIG. 1) as described herein. The operations may startperiodically at 502. At 504, images of a patient may be captured in amedial environment, for example, by a sensing device (e.g., the sensingdevice 100 or 300) or by an existing sensor (e.g., an existing camera)in the medical environment. At 506, a human model may be derived basedon the images of the patient captured at 504. As described herein, sucha human model may be derived by the sensing device and may berepresented by a 2D mesh, a 3D mesh, a 2D contour, a 3D contour, etc.that indicates one or more anatomical or physical characteristics (e.g.,shape, pose, etc.) of the patient as depicted in the captured images.

At 508, a determination may be made regarding whether the patient needsto be positioned for an upcoming medical procedure. If the determinationis that the patient requires positioning, a further determination may bemade at 510 based on the derived human model about whether a currentposition of the patient as indicated by the human model meets therequirements of a protocol designed for the medical procedure. If thecurrent position of the patient meets the requirements, a confirmationmay be provided at 512 to the patient and/or a medical professionaloverseeing the medical procedure. Otherwise, adjustment instructions(e.g., commands) may be provided to the patient at 512 to help thepatient move into the required position. Adjustment instructions (e.g.,control signals) may also be provided to the medical device involved inthe procedure to alter one or more relevant operating parameters (e.g.,height of a scan bed) of the device.

After the operation at 512 or if the determination at 508 is that thepatient does not require positioning, another determination may be madeat 514 regarding whether there are scan images of the patient that needto be analyzed. If the determination is that there are scan images to beanalyzed, the scan images may be analyzed at 516 using the human modelas a reference. For instance, one or more of the scan images may bealigned with the human model and/or with each other based on commonanatomical landmarks identified in the scan images and the human model.The aligned scan images may then be analyzed together to obtain aholistic view of the patient's medical conditions.

After the operation at 516 or if the determination at 514 is that thereare no scan images to be analyzed, another determination may be made at518 regarding whether a target area for scan or treatment needs to belocated. If there is such a target area, the human model may be used tolocate the area and/or provide navigation guidance towards the targetarea, for example, based on body structure information comprised in thehuman model. After the operation at 520 or if the determination at 518is that there is no target area to be located, the set of operations mayend at 522.

For simplicity of explanation, the operations of the medical system aredepicted in FIG. 5 and described herein with a specific order. It shouldbe appreciated, however, that these operations may occur in variousorders, concurrently, and/or with other operations not presented ordescribed herein. Furthermore, it should be noted that not alloperations that the medical system is capable of performing are depictedin FIG. 5 and described herein. It should also be noted that not allillustrated operations may be required to be performed by the medicalsystem.

Further, the medical environment described herein may include a fitnessor rehab facility and the sensing device may be used to monitor and/orguide physical therapy/rehab, training, sports, etc. For example, thesensing device may be used in these settings to track the movements of apatient or athlete (e.g., in real time), compare the movements withguidelines/instructions, and suggest necessary adjustment to improve thetraining or rehab activities.

While this disclosure has been described in terms of certain embodimentsand generally associated methods, alterations and permutations of theembodiments and methods will be apparent to those skilled in the art.Accordingly, the above description of example embodiments does notconstrain this disclosure. Other changes, substitutions, and alterationsare also possible without departing from the spirit and scope of thisdisclosure. In addition, unless specifically stated otherwise,discussions utilizing terms such as “analyzing,” “determining,”“enabling,” “identifying,” “modifying” or the like, refer to the actionsand processes of a computer system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(e.g., electronic) quantities within the computer system's registers andmemories into other data represented as physical quantities within thecomputer system memories or other such information storage, transmissionor display devices.

It is to be understood that the above description is intended to beillustrative, and not restrictive. Many other implementations will beapparent to those of skill in the art upon reading and understanding theabove description. The scope of the disclosure should, therefore, bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

1-20. (canceled)
 21. A portable sensing device, comprising: a housingconfigured to be installed on a device or a structure in a medicalenvironment; one or more sensors configured to be removably attached tothe housing and capture at least one image of a patient in the medicalenvironment; one or more processors configured to be removably attachedto the housing and determine, based on a machine-learning (ML) model andthe at least one image of the patient, one or more human models for thepatient, wherein the ML model is trained to estimate a body shape and aplurality of joint angles of the patient based on the at least one imageof the patient, and wherein the one or more human models are determinedbased on the estimated body shape and the estimated joint angles of thepatient; and one or more communication circuits configured to transmitinformation associated with the one or more human models to a receivingdevice.
 22. The portable sensing device of claim 21, wherein the one ormore processors include a first subset of processors configured todetermine a first human model in accordance with requirements associatedwith a first imaging modality, and wherein the one or more processorsfurther include a second subset of processors configured to determine asecond human model in accordance with requirements associated with asecond imaging modality.
 23. The portable sensing device of claim 22,wherein the first imaging modality is associated with a first one ofX-ray imaging, computer tomography (CT) imaging, or magnetic resonanceimaging (MRI), and wherein the second imaging modality is associatedwith a second one of X-ray imaging, CT imaging, or MRI.
 24. The portablesensing device of claim 22, wherein the housing includes a first slotand a second slot, the first slot configured to host the first subset ofprocessors, the second slot configured to host the second subset ofprocessors.
 25. The portable sensing device of claim 21, wherein the oneor more sensors include a first subset of sensors configured to capturea first image of the patient in accordance with requirements associatedwith a first imaging modality, and wherein the one or more sensorsfurther include a second subset of sensors configured to capture asecond image of the patient in accordance with requirements associatedwith a second imaging modality.
 26. The portable sensing device of claim21, wherein the one or more sensors include a red-green-blue (RGB)sensor configured to capture an RGB image of the patient and a depthsensor configured to capture a depth image of the patient, and whereinthe ML model is trained to estimate the body shape and the plurality ofjoint angles of the patient based on the RGB image and the depth image.27. The portable sensing device of claim 21, wherein the housing isconfigured to be attached to a medical scanner located in the medicalenvironment.
 28. The portable sensing device of claim 21, wherein theone or more human models include a three-dimensional (3D) mesh of thepatient.
 29. The portable sensing device of claim 21, wherein the one ormore human models indicate a position or a pose of the patient before orduring a medical procedure.
 30. The portable sensing device of claim 29,wherein the one or more processors are further configured to determinerelative positions of the patient and a medical device associated withthe medical procedure, and wherein the information transmitted to thereceiving device includes an indication of the relative positions of thepatient and the medical device.
 31. The portable sensing device of claim21, wherein the ML model is trained to extract a plurality of featuresfrom the at least one image of the patient and to estimate the bodyshape and the plurality of joint angles of the patient based on theextracted features.
 32. The portable sensing device of claim 31, whereinthe ML model is implemented through an artificial neural network. 33.The portable sensing device of claim 21, wherein the one or morecommunication circuits are further configured to receive a commandassociated with determining the one or more human models from anexternal device, and wherein the one or more processors are furtherconfigured to determine the one or more human models based on thecommand.
 34. A system, comprising: a portable sensing device, wherein:the portable sensing device comprises one or more sensors and one ormore processors; the one or more sensors are configured to capture atleast one image of a patient in a medical environment; and and the oneor more processors are configured to estimate, using a machine-learning(ML) model, a body shape and a plurality of joint angles of the patientbased on the at least one image of the patient and determine a humanmodel for the patient based on the estimated body shape and theestimated joint angles of the patient; and a processing device, wherein:the processing device is configured to receive information associatedwith the human model from the portable sensing device; and generate,based on the received information, a command for guiding or alerting thepatient or another person, or for controlling a medical device in themedical environment.
 35. The system of claim 34, wherein the portablesensing device includes a housing configured to be installed on a deviceor a structure in the medical environment, and wherein the one or moresensors and the one or more processors of the portable sensing deviceare configured to be removably attached to the housing.
 36. The systemof claim 35, wherein the portable sensing device includes a first subsetof processors configured to determine a first human model in accordancewith requirements associated with a first imaging modality, the portablesensing device further includes a second subset of processors configuredto determine a second human model in accordance with requirementsassociated with a second imaging modality.
 37. The system of claim 36,wherein the housing includes a first slot and a second slot, the firstslot configured to host the first subset of processors, the second slotconfigured to host the second subset of processors.
 38. The system ofclaim 34, wherein the one or more sensors of the portable sensing deviceinclude a red-green-blue (RGB) sensor configured to capture an RGB imageof the patient and a depth sensor configured to capture a depth image ofthe patient, and wherein the ML model is trained to estimate the bodyshape and the plurality of joint angles of the patient based on the RGBimage and the depth image.
 39. The system of claim 34, wherein the humanmodel includes a three-dimensional (3D) mesh of the patient.
 40. Thesystem of claim 34, wherein the processing device is further configuredto determine relative positions of the patient and the medical device inresponse to receiving the information from the portable sensing device,and wherein the command for guiding or alerting the patient or the otherperson, or for controlling the medical device is generated based on therelative positions of the patient and the medical device.